System to collect and visualize software usage metrics

ABSTRACT

Example embodiments involve a metrics collection system for collecting software usage metrics from one or more client devices at deployments. A computer, such as a server configured to execute the metrics collection system, collects software usage metrics (e.g., as a metrics submission from a client device) of the software product at the deployment, identifies a metrics type of the software usage metrics collected, assigns the software usage metrics to a metrics category, and calculates and updates a metrics score of the metrics category, based on the software usage metrics collected.

PRIORITY APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/730,561, filed Dec. 30, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/178,387, filed Jun. 9, 2016, the disclosures ofwhich are incorporated herein in their entirety by reference.

TECHNICAL FIELD

The subject matter disclosed herein relates to graphical user interfacesfor the presentation and visualization of data. In particular, exampleembodiments may relate to machines configured to collect metrics data ofsoftware, and generate and display visualizations of the metrics datawith a specially configured interface.

BACKGROUND

In order to identify bugs and areas that may need improvement insoftware products, software developers may look at usage metrics ofsoftware products at one or more user devices. Usage metrics describewhat features of a software product are used and how those features areused by users of the software products.

BRIEF DESCRIPTION OF THE DRAWINGS

Various ones of the appended drawings merely illustrate exampleembodiments of the present disclosure and are not intended to limit itsscope to the illustrated embodiments. On the contrary, these examplesare intended to cover alternatives, modifications, and equivalents asmay be included within the scope of the disclosure.

FIG. 1 is a network diagram depicting a networked system comprising oneor more application servers in communication with a network-basedmetrics collection system configured for collecting software usagemetrics data from one or more devices, according to some exampleembodiments.

FIG. 2 is a block diagram illustrating various components of the metricscollection system, which is provided as part of the networked system,according to some example embodiments.

FIG. 3 is a flowchart illustrating a method for collecting softwareusage metrics data from a deployed system, and updating a metrics scoreassociated with the software usage metrics data, according to someexample embodiments.

FIG. 4 is a flowchart illustrating a method for causing display of avisualization of software usage metrics data, according to some exampleembodiments.

FIG. 5 is a flowchart illustrating a method for defining a metricsinterval of the metrics collection system, according to some exampleembodiments.

FIG. 6 is a diagram illustrating various interactions between deployedsystems and the metrics collection system, according to some exampleembodiments.

FIG. 7 is an interface diagram illustrating a metrics collectioninterface, according to example embodiments.

FIG. 8 is an interface diagram illustrating a portion of a metricsvisualization interface, according to some example embodiments.

FIG. 9 is an interface diagram illustrating a portion of a metricsvisualization interface, according to some example embodiments.

FIG. 10 is an interface diagram illustrating a portion of a metricsvisualization interface, according to some example embodiments.

FIG. 11 is an interface diagram illustrating a portion of a metricsvisualization interface, according to some example embodiments.

FIG. 12 is an interface diagram illustrating a metrics submissioninterface, according to some example embodiments.

FIG. 13 is an interface diagram illustrating a manual metrics submissionform, according to some example embodiments.

FIG. 14 is a diagrammatic representation of a machine in the exampleform of a computer system within which a set of instructions for causingthe machine to perform any one or more of the methodologies discussedherein may be executed.

DETAILED DESCRIPTION

Reference will now be made in detail to specific example embodiments forcarrying out the inventive subject matter. Examples of these specificembodiments are illustrated in the accompanying drawings, and specificdetails are set forth in the following description in order to provide athorough understanding of the subject matter. It will be understood thatthese examples are not intended to limit the scope of the claims to theillustrated embodiments. On the contrary, they are intended to coversuch alternatives, modifications, and equivalents as may be includedwithin the scope of the disclosure. Examples merely typify possiblevariations. Unless explicitly stated otherwise, components and functionsare optional and may be combined or subdivided, and operations may varyin sequence or be combined or subdivided.

As noted above, usage metrics of software products may be analyzed bysoftware developers to identify bugs and areas which may requireimprovement. In cases where there are multiple software productsexecuting at a large number of client devices, the collection andanalysis of the software usage metrics quickly becomes unmanageable dueto the volume and diversity of the usage metrics collected. For example,usage metrics gathered from a first client device related to a softwareproduct may be dramatically different from usage metrics of the samesoftware product executing at a second client device, due to differencesin the host systems, as well as differences in tasks being executed withthe software product. Therefore, making sense of usage metrics withoutthe aid of computer generated visualizations is time consuming anddifficult—especially when considering that many of the software usagemetrics gathered may pertain to intangible aspects of the softwareproducts themselves. Thus, a system and method to standardize andcollect software usage metrics and to generate and cause display ofvisualizations of the software usage metrics would be an advantage.

Example embodiments involve a metrics collection system for collectingsoftware usage metrics from one or more client devices at deployments.The term “deployments,” as used herein, refers to a group of devicesconfigured to execute a version of a software product. For example, adeployment may include one or more devices configured to execute one ormore distinct or similar products of the same or even a differentversions. A computer, such as a server configured to execute the metricscollection system, collects software usage metrics (e.g., as a metricssubmission from a client device) of the software product at thedeployment, identifies a metrics type of the software usage metricscollected, assigns the software usage metrics to a metrics category, andcalculates and updates a metrics score of the metrics category, based onthe software usage metrics collected.

The software usage metrics collected by the metrics collection systeminclude a rate or frequency with which features of a software productare executed, a number of devices executing the software product at adeployment, a number of deployments executing versions of the softwareproduct, a number of unique users, a number of failed login attempts(e.g., by location, user, or day), a frequency of use of the softwareproduct, a frequency of crashes, bug reports, and performance metricsrelated to a speed or efficiency of actions of the software product. Asa means of standardization, the metrics collection system may include ametrics application that executes at client devices, to quantify andformat metrics submissions for the metrics collection system. The usagemetrics of the metrics submissions may be based on a “Uniform MetricsIdentifier” (UMI) that quantifies, based on what the software productis, what the actual metric collected is, and what the point or durationscale of the metric is, and a value of the metric.

The UMI may comprise three types of information: a group (e.g., thesoftware product the metric is related to); a metric (e.g., what isbeing measured); and a duration (e.g., a timeframe over which themeasurement was made, or an indication if the measurement is just apoint value). For example, based on the UMI, the usage metrics may beformatted as a concatenation of strings associated with the abovecomponents, separated by “:” in the following form:

-   -   <Group>:<Metric>:<Duration>

In some example embodiments, the “<Group” and “<Metric>” component ofthe UMI can be further split out into terms separated by “.” in thefollowing form:

<G-term-1>.<G-term-2>.<G-term-N>:<M-term-1>.<M-term-2>.<M-term-N>:<Duration>

The “<Group>” string indicates what the metric is in reference to. Asused herein, the “<Group>” string identifies a particular softwareproduct (e.g., from among a corpus of products). The “<Group>” portionof the UMI may consist of an arbitrarily nested set of terms thatprovide increasing levels of specificity from left to right. Similarly,the “<Metric>” portion of the UMI may describe what feature of thesoftware product is being measured (e.g., a rate of use of distinctfeatures of the software product, number of unique users, etc.) and mayalso consist of an arbitrarily nested set of terms that provideincreasing levels of specificity from left to right.

The “<Duration>” string indicates a metrics type of the collectedsoftware usage metrics. In general, there are two types of metrics asdiscussed herein: point measurements and duration measurements. A pointmeasurement refers to a metric that is taken at an instant in time(e.g., version number). For point measurements, the “<Duration>”component of the UMI may be omitted entirely. A duration measurementrefers to an observation over a period of time. Duration measurementsare designated in the UMI by appending the associated timeframes as astring. For example, “<Duration>” strings may include “MONTHLY,”“WEEKLY,” “DAILY,” “HOURLY,” and so on.

For each received metrics submission, the metrics collection systemproceeds to identify a metrics type of the usage metrics collected basedon the UMI discussed above. The metrics collection system may thenassign the usage metrics of the metrics submission to an appropriatemetrics category (e.g., issues, engagement, growth, etc.), based on themetrics type and information within the UMI, such as the “<Metric>”string. For example, an administrator of the metrics collection systemmay provide metrics category definitions that assign all metrics relatedto a number of login attempts, a number of program crashes, and a numberof software reboots of a software product at a deployment to a “issues”metrics category, and the administrator may assign a rate of increase inusage, a rate of increase of unique users, and a number of systems thatupgrade to a newer version of the software product to the “growth”metrics category. As the metrics collection system receives metricssubmissions, the metrics collection system may categorize the softwareusage metrics data of the metrics submissions into categories based onthe corresponding UMI, and category definitions provided by anadministrator of the metrics collection system. The metrics collectionsystem calculates a metrics score of each metrics category based on theusage metrics collected.

In some example embodiments, the metrics collection system generates andcauses display of a graphical user interface at a client device toreceive visualization requests. For example, the graphical userinterface displayed at the client device may include a set of menusconfigured to receive visualization requests, wherein the visualizationrequests include an indication of a metrics category, a deployment, atimeframe or duration, and a visualization type. In response toreceiving the visualization request, the metrics collection systemgenerates and causes display of the visualization within the graphicaluser interface at the client device.

FIG. 1 is a network diagram illustrating a network environment 100suitable for operating a metrics collection system 150, according tosome example embodiments. A networked system 102 provides server-sidefunctionality, via a network 104 (e.g., an intranet, the Internet, or aWide Area Network (WAN)), to one or more clients such as a client device110 (operable by a user 106) and a deployment 130. FIG. 1 illustrates aweb client 112 and a metrics application 114 executing on the clientdevice 110.

An Application Program Interface (API) server 120 and a web server 122are coupled to, and provide programmatic and web interfaces respectivelyto, one or more application servers 140. The application servers 140host the metrics collection system 150. The application servers 140 are,in turn, shown to be coupled to one or more database servers 124 thatfacilitate access to one or more databases 126.

The metrics collection system 150 performs operations that includereceiving metrics submissions that include software usage metrics fromthe deployment 130 and the client device 110, identifying a metrics typeof the software usage metrics, categorizing the software usage metrics,and generating and causing display of a visualization of the softwareusage metrics within a graphical user interface, for the networkedsystem 102. The deployment 130 may be or include a database (e.g.,similar to the database 126). In some example embodiments, thedeployment 130 includes a web server machine operated by a third party(e.g., an entity distinct from the metrics collection system 150).

As shown, the network environment 100 includes the client device 110 incommunication with the networked system 102 over the network 104. Thenetworked system 102 communicates and exchanges data with the clientdevice 110 that pertains to various functions and aspects associatedwith the networked system 102 and its users. Likewise, the client device110, which may be any of a variety of types of devices that include atleast a display, a processor, and communication capabilities thatprovide access to the network 104 (e.g., a smart phone, a tabletcomputer, a personal digital assistant (PDA), a personal navigationdevice (PND), a handheld computer, a desktop computer, a laptop ornetbook, or a wearable computing device), may be operated by the user106 (e.g., a person) to exchange data with the networked system 102 overthe network 104.

The client device 110 communicates with the network 104 via a wired orwireless connection. For example, one or more portions of the network104 may comprise an ad hoc network, an intranet, an extranet, a VirtualPrivate Network (VPN), a Local Area Network (LAN), a wireless LAN(WLAN), a WAN, a wireless WAN (WWAN), a Metropolitan Area Network (MAN),a portion of the Internet, a portion of the Public Switched TelephoneNetwork (PSTN), a cellular telephone network, a wireless network, aWireless Fidelity (Wi-Fi®) network, a Worldwide Interoperability forMicrowave Access (WiMax) network, another type of network, or anysuitable combination thereof.

In various embodiments, the data exchanged between the client device 110and the networked system 102 may involve user-selected functionsavailable through one or more user interfaces (UIs). The UIs may bespecifically associated with the web client 112 (e.g., a browser) or themetrics application 114, executing on the client device 110, and incommunication with the networked system 102. In further embodiments, theUIs may be served to the client device 110 through an encryptedtransport layer (i.e., SSL.TLS).

Turning specifically to the networked system 102, the web server 122 iscoupled to (e.g., via wired or wireless interfaces), and provides webinterfaces to, the application server 140. In some embodiments, themetrics collection system 150 runs and executes on the applicationserver 140, while in other embodiments, the application server 140provides the client device 110 with a set of instructions (e.g.,computer-readable code) that causes the web client 112 and the metricsapplication 114 of the client device 110 to execute and run the metricscollection system 150.

FIG. 2 is a block diagram illustrating various components of the metricscollection system 150, which is provided as part of the networked system102, consistent with some embodiments. To avoid obscuring the inventivesubject matter with unnecessary detail, various functional components(e.g., modules and engines) that are not germane to conveying anunderstanding of the inventive subject matter have been omitted fromFIG. 2 . However, a skilled artisan will readily recognize that variousadditional functional components may be supported by the metricscollection system 150 to facilitate additional functionality that is notspecifically described herein.

As is understood by skilled artisans in the relevant computer arts, eachfunctional component (e.g., module) illustrated in FIG. 2 may beimplemented using hardware (e.g., a processor of a machine) or acombination of logic (e.g., executable software instructions) andhardware (e.g., memory and a processor of a machine) for executing thelogic. Furthermore, the various functional components depicted in FIG. 2may reside on a single computer (e.g., a laptop), or may be distributedacross several computers in various arrangements such as cloud-basedarchitectures. Moreover, it shall be appreciated that while thefunctional components (e.g., modules) of FIG. 2 are discussed in thesingular sense, in other embodiments, multiple instances of one or moreof the modules may be employed.

The metrics collection system 150 is shown as including a collectionmodule 210, a categorization module 220, a scoring module 230, and avisualization module 240, all configured to communicate with each other(e.g., via a bus, shared memory, a switch, or APIs).

The collection module 210 obtains metrics submissions from multiple datasources. Data sources for metrics submission data that includes softwareusage metrics include the deployment 130, as well as the client device110. The deployment 130 may comprise a set of devices executing one ormore software products. The metrics submissions include software usagemetrics, formatted based on the UMI (as discussed above).

The categorization module 220 identifies a data type of the softwareusage metrics of the metrics submission based on the UMI, and assignsthe software usage metrics to a metrics category. Metrics categories mayinclude, but are not limited to, “issues,” “growth,” “engagement,” and“performance.” In some example embodiments, an administrator of themetrics collection system 150 may provide additional metrics categorydefinitions to the categorization module 220. Metrics categorydefinitions include a metrics category identifier, and a correspondinglist of features from the UMI for the metrics category identifier. Inthis way, an administrator of the metrics collection system 150 maydefine new metrics categories, or add features to existing metricscategories.

The scoring module 230 calculates a metrics score of each metricscategory of each deployment, each individual system, and each softwareproduct. The score calculated by the scoring module 230 is based on thesoftware usage metrics collected by the collection module 210. Forexample, the scoring can be done on an aggregated level to aggregatemetrics themselves. Consider an example embodiment in which a deployment(e.g., deployment A) includes two devices (e.g., first device and seconddevice) that are running a software product (e.g., Software Product A).The first device may report ten unique users in a particular week andthe second device may report thirty unique users in the same week, whereeach metric (i.e., unique user logins) is an aggregate of login eventson each device. The scoring module 230 may thereby apply a scoringcalculation to quantify all deployments running “Software Product A,” orvice versa, all software products installed on Deployment A itself, inorder to calculate a state/score for the quantification. In someembodiments, the scoring calculation may manifest as an algorithm thatcauses the scoring module 230 to count all instances of “SoftwareProduct A” running with more than fifteen users, and give the clientdevice one point, then sum the points up for all devices to get a score.

The visualization module 240 receives visualization requests from one ormore client devices. The visualization requests include indications of avisualization type (e.g., bar graph), a metrics category or feature, anda time period. The visualization module 240 generates and causes displayof a visualization at the client device (e.g., client device 110) basedon the visualization request.

FIG. 3 is a flowchart illustrating operations of the metrics collectionsystem 150 in performing a method 300 for collecting software usagemetrics from a data source (e.g., a deployment), categorizing the usagemetrics, and updating a metrics score associated with the metricscategory corresponding to the usage metrics, according to some exampleembodiments. The method 300 is embodied in computer-readableinstructions for execution by one or more processors, such that theoperations of the method 300 are performed in part or in whole by themetrics collection system 150; accordingly, the method 300 is describedbelow by way of example with reference thereto. However, it shall beappreciated that at least some of the operations of the method 300 maybe deployed on various other hardware configurations, and the method 300is not intended to be limited to the metrics collection system 150.

At operation 310, the collection module 210 receives a metricssubmission from a data source. The metrics submission may be deliveredto the metrics collection system 150 as an e-mail, as a manual usersubmission, or through a distributed queue messaging service. Forexample, manual user submissions may be accomplished via API through afront end GUI. Message queues provide an asynchronous communicationsprotocol, meaning that the sender and receiver of the message do notneed to interact with the message queue at the same time. Messagesplaced onto the queue are stored until the recipient retrieves them.Message queues have implicit or explicit limits on the size of data thatmay be transmitted in a single message and the number of messages thatmay remain outstanding on the queue.

As an example of the forgoing operation, the metrics application 114 maycause display of a graphical user interface configured to receive andtransmit metrics submissions at a client device 110. The user 106 of theclient device 110 may submit a metrics submission (e.g., a UMI) to themetrics collection system 150 through the interface. The software usagemetrics are then collected automatically by the metrics application 114,and delivered to the metrics collection system 150 through the network104. For example, the metrics application 114 may monitor variousmetrics features of a software product (or multiple software products)executing on the client device 110 (or at the deployment 130). Themetrics application 114 may then deliver the software usage metricscollected to the collection module 210 as a metrics submission (e.g.,UMI).

At operation 320, the categorization module 220 identifies a metricstype of the software usage metrics within the metrics submission, basedon a UMI. As discussed above, the UMI includes a field indicating ametrics type of the software usage metrics collected. The categorizationmodule 220 parses the metrics submission received by the collectionmodule 210 to identify the metrics type.

At operation 330, the categorization module 220 categorizes the softwareusage metrics data of the metrics submission based on the UMI. Asdiscussed above, the UMI includes a field that identifies the specificfeature being measured. The categorization module 220 accesses a list ofmetrics category definitions, and based on the metrics categorydefinitions and the UMI, categorizes the software usage metrics data.The categorization module 220 may, in some instances, assign thesoftware usage metrics data of the metrics submission to multiplemetrics categories.

At operation 340, the scoring module 230 calculates a metrics score ofthe metrics category (or categories). The metrics score is based on thesoftware usage metrics values collected by the collection module 210.

FIG. 4 is a flowchart illustrating a method 400 for generating andcausing display of a visualization of software usage metrics data at aclient device 110, according to some example embodiments. As shown inFIG. 4 , one or more operations 410 and 420 may be performed as part(e.g., a precursor task, a subroutine, or a portion) of operation 340,in which the scoring module 230 updates a metrics score of the metricscategories based on metrics submissions collected by the collectionmodule 210, according to some example embodiments.

At operation 410, the visualization module 240 receives a visualizationrequest from a client device (e.g., client device 110). Thevisualization request may include a set of visualization criteria, suchas a visualization type, as well as an indication of a deployment orclient device, metrics category, feature to visualize, and softwareproduct identifier. In some example embodiments, the visualizationmodule 240 causes display of a visualization interface at a clientdevice (e.g., the client device 110). A user of the client device 110may provide the visualization criteria through one or more interfaceelements of the visualization interface. The interface elements mayinclude drop down menus, text fields, and user selectable icons.

At operation 420, in response to receiving the visualization request,the visualization module 240 generates and causes display of avisualization of the software usage metrics data at the client device110. Examples of visualizations generated and displayed by thevisualization module 240 can be seen in FIGS. 8-11 . In someembodiments, the visualization module 240 receives a selection of avisualization type (e.g., bar graph), and generates a visualizationbased on the selection. In some embodiments, the visualization module240 may select a visualization type based on elements of thevisualization request itself, such as the deployment or productselected.

FIG. 5 is a flowchart illustrating a method 500 for defining a metricsinterval to receive metrics submissions, according to some exampleembodiments. As shown in FIG. 5 , one or more operations 510 and 520 maybe performed as part (e.g., a precursor task, a subroutine, or aportion) of operation 310, in which the collection module 210 receives ametrics submission from a data source, according to some exampleembodiments.

At operation 510, the collection module 210 receives an automatedmetrics interval that includes a rate at which to access a data source(e.g., deployment 130, client device 110). For example, to receive theautomated metrics interval, the collection module 210 may cause displayof a metrics interface to set up automated metrics submissions (as seenin FIG. 12 ) at a client device. A user may provide a metrics intervaldefinition through the metrics interface to define an automated metricsinterval at which to collect and transmit metrics submissions to themetrics collection system 150. For example, the metrics intervaldefinition may include “weekly,” as well as “daily.” In some exampleembodiments, the metrics interval definition includes a feature, asoftware product, a deployment, and a rate at which to collect andprovide metrics submissions. In some embodiments, the metrics intervaldefinition configures the metrics application 114 to collect andtransmit metrics submissions at the defined rate. In some embodiments,the metrics interval definition configures the collection module 210 toquery a data source for the requested metrics based on the metricsinterval definition.

At operation 520, based on the metrics interval definition received fromthe client device 110 through the metrics interface, the collectionmodule 210 delivers a metrics request to the data source (e.g.,deployment 130, client device 110). The metrics request includes afeature, a metrics type, a software product, and a period of time overwhich to retrieve software usage metrics data. Based on the metricsrequest, the data source provides a metrics submission to the collectionmodule 210.

FIG. 6 is a system diagram 600 illustrating various interactions betweendeployments 610 and the metrics collection system 150, according to someexample embodiments. As seen in FIG. 6 , the deployments 610 may includeone or more deployments (e.g., deployment A, deployment B, deploymentC), each comprising one or more host systems (e.g., host 1, host 2). Thehost systems may include the client device 110, and the deployments 610may include deployment 130, as seen in FIG. 1 . In some exampleembodiments, each host system may contain one to N metrics applicationsthat may or may not be unique from one another.

The deployments 610 comprise data sources of software usage metrics datafor the metrics collection system 150. For example, the deployment 130of FIG. 6 may comprise a grouping of systems configured to execute asoftware platform consisting of one or more products. The systems may begrouped based on being a part of the same company or organization, teamwithin a company, or building.

Software usage metrics data in the form of metrics submissions flow fromthe deployments 610, through the network 104, and into the metricscollection system 150 based on the methods 300, 400, and 500 discussedin FIGS. 3-5 . The metrics collection system 150 receives the metricssubmissions, and stores the software usage metrics data within thedatabase 126.

FIG. 7 is an interface diagram illustrating a metrics collectioninterface 700, according to example embodiments. As shown, the metricscollection interface 700 includes a set of interface elements 710, 720,and 730 configured to receive user inputs to generate and cause displayof visualizations of software usage metrics data at a client device 110.

The interface element 710 allows users to submit visualization requestsfor software usage metrics associated with a deployment. For example, adeployment may execute one or more software products on a number ofsystems associated with the deployment. A user may select the interfaceelement 710, and in response, be presented with a selectable list ofdeployments (e.g., deployment 130). By selecting a deployment from thelist, the user may be presented with one or more visualization optionsin order to generate and cause display of a visualization of softwareusage metrics associated with the selected deployment.

The interface element 720 allows users to submit to visualizationrequests for software usage metrics associated with a software productacross multiple deployments. For example, a single software product maybe used in multiple deployments. A user may choose to visualize how thesoftware product is being used across the multiple deployments byselecting the interface element 720. A software product may include acomputer program executing at a device (e.g., client device 110).

The interface element 730 allows users to view software usage metrics ofthe metrics collection system 150, and the metrics application 114. Forexample, a user may select the interface element 730, and in response bepresented with an interface to view software usage metrics of themetrics collection system 150 and metrics application 114.

FIG. 8 is an interface diagram illustrating a metrics visualizationinterface 800, according to example embodiments. As shown, the metricsvisualization interface 800 includes a group selection menu 810configured to receive a selection of a group identifier (e.g.,deployment A), feature identifiers 820 and 830, and a product identifier840. The metrics visualization interface 800 may be presented at aclient device 110 in response to a selection of the interface element720 of FIG. 7 .

Selection of the group selection menu 810 may cause display of a listingof available group identifiers. Each group identifier may correspond toa unique deployment that comprises one or more systems executingsoftware products. For example, if group identifier “deployment A” isselected from the group selection menu 810, the metrics collectionsystem 150 accesses the database 126 to retrieve and present a list ofsoftware products executing on devices associated with deployment A. Forexample, deployment A may have one or more associated devices whichexecute products 1-5, as show in FIG. 8 .

If a product identifier is selected from among the list of productidentifiers (e.g., product identifier 840), the metrics collectionsystem 150 causes display of visualizations 850 and 860, based on thefeature identifiers 820 and 830. For example, the visualization 850 maydepict a visualization of software usage metrics related to the featureidentifier 820 (total document views per week) of software product 1. Ifthe product identifier 840 is selected, the metrics visualizationinterface 800 updates to include visualizations based on software usagedata that corresponds to the selected product identifier (e.g., productidentifier 840).

FIG. 9 is an interface diagram illustrating a metrics visualizationinterface 900, according to example embodiments. As shown, the metricsvisualization interface 900 includes a product identifier 910, metricscategory identifiers 920, 930, and 940, and a visualization 950. Themetrics visualization interface 900 may be presented at a client device110 in response to a selection of the interface element 720 of FIG. 7 .

The metrics visualization interface 900 presents softwareproduct-specific metrics (of software product 1), across all deploymentswhich are executing the software product. As shown in FIG. 9 , themetrics visualization interface 900 includes a presentation of themetrics category identifiers 920, 930, and 940. As discussed above, themetrics categories may be defined by an administrator of the metricscollection system 150, by methods discussed above in reference to thecategorization module 220 depicted in FIG. 2 .

A user may select a metrics category identifier (e.g., metrics categoryidentifier 920), and in response the metrics collection system 150 mayupdate the metrics visualization interface 900 to display visualizationsgenerated based on software usage metrics of features related to theselected metrics category.

For example, if the user selects the metrics category identifier 930,the metrics visualization interface 900 may update to displayvisualizations of features 1020, 1030, and 1040 of FIG. 10 , based onthe metrics category identifier 930 (e.g., engagement metrics). As seenin FIG. 10 , the metrics category identifier 930 includes features 1020(e.g., unique users), 1030 (e.g., document views), and 1040 (documentviews per user). As explained above, the features 1020, 1030, and 1040are associated with the metrics category identifier 930 by anadministrator of the metrics collection system 150. Similarly, if theuser selects the metrics category identifier 940, the metricsvisualization interface 900 updates to display a visualization 1110 asshown in FIG. 11 , based on software usage metrics data of featuresassociated with the pain metrics category.

FIG. 12 is an interface diagram illustrating a metrics collectioninterface 1200, according to example embodiments. As shown, the metricscollection interface 1200 includes a set of interface elements 1210,1220, and 1230 configured to receive user inputs to provide metricssubmissions to the metrics collection system 150, according to exampleembodiments.

Selection of the interface element 1210 causes the metrics collectioninterface 1200 to display a manual metrics submission form 1300, as seenin FIG. 13 . A user may provide metrics submissions manually to themetrics collection system 150, through the manual metrics submissionform 1300. The metrics submission may thereby be delivered to themetrics collection system 150 electronically, via email or other similarelectronic delivery methods.

Selection of the interface element 1220 causes the metrics collectioninterface 1200 to display an interface to receive automated metrics,according to the method 500 depicted in FIG. 5 . Upon receiving theautomated metrics instructions through the interface, the metricscollection system 150 configures itself, or in some embodiments a clientdevice 110 executing a metrics application 114, to retrieve metricssubmissions of requested features at defined intervals.

Selection of the interface element 1230 causes the metrics collectioninterface 1200 to display one or more interface elements to viewsoftware usage metrics existing within the metrics collection system150, at the database 126.

FIG. 13 is an interface diagram illustrating a manual metrics submissionform 1300, to manually submit a metrics submission to the metricscollection system 150, according to example embodiments. The manualmetrics submission form 1300 includes: a deployment menu 1310; interfaceelements 1320; 1330, and 1340 to receive metrics submission details asuser inputs; a submission result indicator 1350; and a display ofhistorical data 1360, according to example embodiments.

A user 106 of the client device 110, configured to display the manualmetrics submission form 1300, may provide metrics submission detailsthrough the interface elements 1320, 1330, and 1340. The interfaceelements may correspond to metrics submission information such as a dateof the software usage metrics being submitted (e.g., 1320), a version ofthe software product which the software usage metrics are associatedwith (e.g., 1330), and software usage metrics features, such as “uniqueweekly logins,” (e.g., 1340). Upon receiving the metrics submissionthrough the manual metrics submission form 1300, the submission resultindicator 1350 updates to display a status of the submission (e.g.,success, failed, etc.). The display of historical data 1360 may alsoupdate to include the metrics submitted.

FIG. 14 is a diagrammatic representation of a machine 1400 in theexample form of a computer system within which a set of instructions forcausing the machine to perform any one or more of the methodologiesdiscussed herein may be executed. Specifically, FIG. 14 shows adiagrammatic representation of the machine 1400 in the example form of asystem, within which instructions 1402 (e.g., software, a program, anapplication, an applet, an app, a driver, or other executable code) forcausing the machine 1400 to perform any one or more of the methodologiesdiscussed herein may be executed. For example, the instructions 1402include executable code that causes the machine 1400 to execute themethods illustrated in FIGS. 3-5 . In this way, these instructions 1402transform the general, non-programmed machine into a particular machineprogrammed to carry out the described and illustrated functions in themanner described herein. The machine 1400 may operate as a standalonedevice or may be coupled (e.g., networked) to other machines.

By way of non-limiting example, the machine 1400 may comprise orcorrespond to a television, a computer (e.g., a server computer, aclient computer, a personal computer (PC), a tablet computer, a laptopcomputer, or a netbook), a set-top box (STB), a personal digitalassistant (PDA), an entertainment media system (e.g., an audio/videoreceiver), a cellular telephone, a smart phone, a mobile device, awearable device (e.g., a smart watch), a portable media player, or anymachine capable of outputting audio signals and capable of executing theinstructions 1402, sequentially or otherwise, that specify actions to betaken by the machine. Further, while only a single machine 1400 isillustrated, the term “machine” shall also be taken to include acollection of machines 1400 that individually or jointly execute theinstructions 1402 to perform any one or more of the methodologiesdiscussed herein.

The machine 1400 may include processors 1404, a memory/storage 1432,memory 1406, a storage unit 1408, and I/O components 1410, which may beconfigured to communicate with each other such as via a bus 1412. In anexample embodiment, the processors 1404 (e.g., a central processing unit(CPU), a reduced instruction set computing (RISC) processor, a complexinstruction set computing (CISC) processor, a graphics processing unit(GPU), a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a radio-frequency integrated circuit (RFIC),another processor, or any suitable combination thereof) may include, forexample, a processor 1414 and a processor 1416 that may execute theinstructions 1402. The term “processor” is intended to includemulti-core processors that may comprise two or more independentprocessors (sometimes referred to as “cores”) that may executeinstructions contemporaneously. Although FIG. 14 shows multipleprocessors, the machine 1400 may include a single processor with asingle core, a single processor with multiple cores (e.g., a multi-coreprocessor), multiple processors with a single core, multiple processorswith multiples cores, or any combination thereof.

The memory 1406 (e.g., a main memory or other memory storage) and thestorage unit 1408 are both accessible to the processors 1404 such as viathe bus 1412. The memory 1406 and the storage unit 1408 store theinstructions 1402 embodying any one or more of the methodologies orfunctions described herein. In some embodiments, the database 126resides on the storage unit 1408. The instructions 1402 may also reside,completely or partially, within the memory 1406, within the storage unit1408, within at least one of the processors 1404 (e.g., within theprocessor's cache memory), or any suitable combination thereof, duringexecution thereof by the machine 1400. Accordingly, the memory 1406, thestorage unit 1408, and the memory of the processors 1404 are examples ofmachine-readable media.

As used herein, “machine-readable medium” means a device able to storeinstructions and data temporarily or permanently and may include, butnot be limited to, random-access memory (RAM), read-only memory (ROM),buffer memory, flash memory, optical media, magnetic media, cachememory, other types of storage (e.g., erasable programmable read-onlymemory (EEPROM)), or any suitable combination thereof. The term“machine-readable medium” should be taken to include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) able to store the instructions 1402. Theterm “machine-readable medium” shall also be taken to include anymedium, or combination of multiple media, that is capable of storinginstructions (e.g., instructions 1402) for execution by a machine (e.g.,machine 1400), such that the instructions, when executed by one or moreprocessors of the machine (e.g., processors 1404), cause the machine toperform any one or more of the methodologies described herein (e.g.,method 400). Accordingly, a “machine-readable medium” refers to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-readable medium” excludes signals per se.

Furthermore, the “machine-readable medium” is non-transitory in that itdoes not embody a propagating signal. However, labeling the tangiblemachine-readable medium as “non-transitory” should not be construed tomean that the medium is incapable of movement—the medium should beconsidered as being transportable from one real-world location toanother. Additionally, since the machine-readable medium is tangible,the medium may be considered to be a machine-readable device.

The I/O components 1410 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 1410 that are included in a particular machine will depend onthe type of machine. For example, portable machines such as mobilephones will likely include a touch input device or other such inputmechanisms, while a headless server machine will likely not include sucha touch input device. It will be appreciated that the I/O components1410 may include many other components that are not specifically shownin FIG. 14 . The I/O components 1410 are grouped according tofunctionality merely for simplifying the following discussion and thegrouping is in no way limiting. In various example embodiments, the I/Ocomponents 1410 may include input components 1418 and output components1420, as well as biometric components 1456. The input components 1418may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-opticalkeyboard, or other alphanumeric input components), point based inputcomponents (e.g., a mouse, a touchpad, a trackball, a joystick, a motionsensor, or other pointing instruments), tactile input components (e.g.,a physical button, a touch screen that provides location and/or force oftouches or touch gestures, or other tactile input components), audioinput components, and the like. The output components 1420 may includevisual components (e.g., a display such as a plasma display panel (PDP),a light emitting diode (LED) display, a liquid crystal display (LCD), aprojector, or a cathode ray tube (CRT)), acoustic components (e.g.,speakers), haptic components (e.g., a vibratory motor, resistancemechanisms), other signal generators, and so forth.

Communication may be implemented using a wide variety of technologies.The I/O components 1410 may include communication components 1422operable to couple the machine 1400 to a network 1424 or devices 1426via a coupling 1428 and a coupling 1430, respectively. For example, thecommunication components 1422 may include a network interface componentor another suitable device to interface with the network 1424. Infurther examples, the communication components 1422 may include wiredcommunication components, wireless communication components, cellularcommunication components, near field communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), WiFi® components,and other communication components to provide communication via othermodalities. The devices 1426 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUniversal Serial Bus (USB)).

Modules, Components and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A hardware module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., a standalone, client, or server computersystem) or one or more hardware modules of a computer system (e.g., aprocessor or a group of processors) may be configured by software (e.g.,an application or application portion) as a hardware module thatoperates to perform certain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field-programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner and/or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses that connect the hardware modules). In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment, or a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), with these operations being accessiblevia a network (e.g., the Internet) and via one or more appropriateinterfaces (e.g., APIs).

Electronic Apparatus and System

Example embodiments may be implemented in digital electronic circuitry,or in computer hardware, firmware, or software, or in combinations ofthem. Example embodiments may be implemented using a computer programproduct, for example, a computer program tangibly embodied in aninformation carrier, for example, in a machine-readable medium forexecution by, or to control the operation of, data processing apparatus,for example, a programmable processor, a computer, or multiplecomputers.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a standalone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site, or distributed across multiple sites andinterconnected by a communication network.

In example embodiments, operations may be performed by one or moreprogrammable processors executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry(e.g., an FPGA or an ASIC).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Inembodiments deploying a programmable computing system, it will beappreciated that both hardware and software architectures meritconsideration. Specifically, it will be appreciated that the choice ofwhether to implement certain functionality in permanently configuredhardware (e.g., an ASIC), in temporarily configured hardware (e.g., acombination of software and a programmable processor), or in acombination of permanently and temporarily configured hardware may be adesign choice. Below are set out hardware (e.g., machine) and softwarearchitectures that may be deployed, in various example embodiments.

Language

Although the embodiments of the present disclosure have been describedwith reference to specific example embodiments, it will be evident thatvarious modifications and changes may be made to these embodimentswithout departing from the broader scope of the inventive subjectmatter. Accordingly, the specification and drawings are to be regardedin an illustrative rather than a restrictive sense. The accompanyingdrawings that form a part hereof show, by way of illustration, and notof limitation, specific embodiments in which the subject matter may bepracticed. The embodiments illustrated are described in sufficientdetail to enable those skilled in the art to practice the teachingsdisclosed herein. Other embodiments may be used and derived therefrom,such that structural and logical substitutions and changes may be madewithout departing from the scope of this disclosure. This DetailedDescription, therefore, is not to be taken in a limiting sense, and thescope of various embodiments is defined only by the appended claims,along with the full range of equivalents to which such claims areentitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent, to those of skill inthe art, upon reviewing the above description.

All publications, patents, and patent documents referred to in thisdocument are incorporated by reference herein in their entirety, asthough individually incorporated by reference. In the event ofinconsistent usages between this document and those documents soincorporated by reference, the usage in the incorporated referencesshould be considered supplementary to that of this document; forirreconcilable inconsistencies, the usage in this document controls.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. In the appended claims, the terms “including” and“in which” are used as the plain-English equivalents of the respectiveterms “comprising” and “wherein.” Also, in the following claims, theterms “including” and “comprising” are open-ended; that is, a system,device, article, or process that includes elements in addition to thoselisted after such a term in a claim is still deemed to fall within thescope of that claim.

What is claimed is:
 1. A system comprising: one or more processors; anda memory comprising instructions which, when executed by the one or moreprocessors, cause the one or more processors to perform operationscomprising: assigning a metrics submission interval to an identifierassociated with a deployment that comprises a plurality of clientdevices; querying each of the plurality of client devices based on themetrics submission interval assigned to the identifier associated withthe deployment; causing a client device from among the plurality ofclient devices to generate a metrics submission based on the metricssubmission interval assigned to the identifier associated with thedeployment, the metrics submission corresponding with a metricscategory; receiving, from the client device, a metrics submission thatcomprises software usage data; generating a score associated with themetrics category, the score comprising an aggregate value based on atleast the metrics submission received from the client device; causingdisplay of a visualization of at least the metrics submission, thevisualization including the score.
 2. The system of claim 1, wherein thecausing display of the visualization of the metrics submission includes:determining a visualization type based on an attribute of the metricssubmission; and causing display of the visualization based on thevisualization type.
 3. The system of claim 1, wherein the metricssubmission comprises a duration string that indicates a metrics type ofthe metrics submission.
 4. The system of claim 3, wherein the causingdisplay of the visualization of the metrics submission includes: causingdisplay of the visualization of the metrics submission based on themetrics type of the metrics submission.
 5. The system of claim 1,wherein the generating the score of the metrics category that comprisesthe aggregate value further comprises: applying a scoring calculationthat comprises of an algorithm that counts all instances of a softwareapplication running with a plurality of users; assigning a point to eachclient device running the software application running with theplurality of users; and summing, using the algorithm, the points for theplurality of client devices to generate the score.
 6. The system ofclaim 1, wherein the metrics submission further comprises indications ofone or more of: a frequency of use of a software application executed bythe client device; a number of unique users of the software applicationat the client device; and a list of actions executed with the softwareapplication at the client device.
 7. A non-transitory machine-readablestorage medium comprising instructions that, when executed by one ormore processors of a machine, cause the machine to perform operationscomprising: assigning a metrics submission interval to an identifierassociated with a deployment that comprises a plurality of clientdevices; querying each of the plurality of client devices based on themetrics submission interval assigned to the identifier associated withthe deployment; causing a client device from among the plurality ofclient devices to generate a metrics submission based on the metricssubmission interval assigned to the identifier associated with thedeployment, the metrics submission corresponding with a metricscategory; receiving, from the client device, a metrics submission thatcomprises software usage data; generating a score associated with themetrics category, the score comprising an aggregate value based on atleast the metrics submission received from the client device; causingdisplay of a visualization of at least the metrics submission, thevisualization including the score.
 8. The non-transitorymachine-readable storage medium of claim 7, wherein the causing displayof the visualization of the metrics submission includes: determining avisualization type based on an attribute of the metrics submission; andcausing display of the visualization based on the visualization type. 9.The non-transitory machine-readable storage medium of claim 7, whereinthe metrics submission comprises a duration string that indicates ametrics type of the metrics submission.
 10. The non-transitorymachine-readable storage medium of claim 9, wherein the causing displayof the visualization of the metrics submission includes: causing displayof the visualization of the metrics submission based on the metrics typeof the metrics submission.
 11. The non-transitory machine-readablestorage medium of claim 7, wherein the generating the score of themetrics category that comprises the aggregate value further comprises:applying a scoring calculation that comprises of an algorithm thatcounts all instances of a software application running with a pluralityof users; assigning a point to each client device running the softwareapplication running with the plurality of users; and summing, using thealgorithm, the points for the plurality of client devices to generatethe score.
 12. The non-transitory machine-readable storage medium ofclaim 7, wherein the metrics submission further comprises indications ofone or more of: a frequency of use of a software application executed bythe client device; a number of unique users of the software applicationat the client device; and a list of actions executed with the softwareapplication at the client device.
 13. A method comprising: assigning ametrics submission interval to an identifier associated with adeployment that comprises a plurality of client devices; querying eachof the plurality of client devices based on the metrics submissioninterval assigned to the identifier associated with the deployment;causing a client device from among the plurality of client devices togenerate a metrics submission based on the metrics submission intervalassigned to the identifier associated with the deployment, the metricssubmission corresponding with a metrics category; receiving, from theclient device, a metrics submission that comprises software usage data;generating a score associated with the metrics category, the scorecomprising an aggregate value based on at least the metrics submissionreceived from the client device; causing display of a visualization ofat least the metrics submission, the visualization including the score.14. The method of claim 13, wherein the causing display of thevisualization of the metrics submission includes: determining avisualization type based on an attribute of the metrics submission; andcausing display of the visualization based on the visualization type.15. The method of claim 13, wherein the metrics submission comprises aduration string that indicates a metrics type of the metrics submission.16. The method of claim 15, wherein the causing display of thevisualization of the metrics submission includes: causing display of thevisualization of the metrics submission based on the metrics type of themetrics submission.
 17. The method of claim 13, wherein the generatingthe score of the metrics category that comprises the aggregate valuefurther comprises: applying a scoring calculation that comprises of analgorithm that counts all instances of a software application runningwith a plurality of users; assigning a point to each client devicerunning the software application running with the plurality of users;and summing, using the algorithm, the points for the plurality of clientdevices to generate the score.